Agent Mode for “Serious” Developers: Autonomous Coding with VS Code

Session Date: May 21, 2025
Duration: 15 minutes
Venue: Build 2025 Conference - DEM519
Speaker: Devin Valenciano (Senior Product Manager, VS Code Team, Microsoft)
Link: Microsoft Build 2025 Session DEM519

Table of Contents

Introduction and Session Overview

Session Opening

Timeframe: 00:00:00 - 00:00:29
Duration: 29s
Speaker: Devin Valenciano

Devin Valenciano introduces himself as an Associate Product Manager (APM) on the VS Code team, setting the stage for a demonstration that challenges traditional perceptions about who can be a “serious developer.” The session’s core premise revolves around democratizing advanced development capabilities through Agent Mode.

“My name is Devin Valenciano… I am APM on the VS Code team. And like Leslie just said, I’m here to talk about Agent mode for serious developers.”

Agent Mode Concept

Timeframe: 00:00:29 - 00:01:09
Duration: 40s
Speaker: Devin Valenciano

The fundamental introduction to Agent Mode establishes it as more than traditional AI assistance—it’s positioned as an autonomous coding partner capable of iterative problem-solving and independent progress.

“Agent mode’s this really cool thing that’s built to the copilot that allows you to solve bigger problems. It’s this autonomous coding assistant that can iterate on its own progress.”

Key Capabilities Introduced:

  • Autonomous iteration - Independent problem-solving progression
  • Complex problem handling - Multi-step, interconnected challenges
  • Role democratization - Enabling non-experts to achieve expert-level results

Democratization of Development

Timeframe: 00:00:21 - 00:00:29
Duration: 8s
Speaker: Devin Valenciano

A crucial philosophical statement that frames the entire demonstration: Agent Mode as an equalizing force in technical development, allowing product managers and non-technical professionals to engage with complex programming challenges.

“One of the coolest things it does is that it lets APM like me pretend to be a serious developer for a little bit or a serious data scientist as we’re about to show.”

Real-World Challenge Setup

Kaggle Competition Introduction

Timeframe: 00:00:29 - 00:00:47
Duration: 18s
Speaker: Devin Valenciano

The session establishes credibility by selecting a genuine competitive programming challenge rather than a contrived demo, positioning Kaggle as the data science equivalent of LeetCode for algorithmic challenges.

“So the example that I’m going to run you all through is this real piece of work. So this is what’s called a Cagle competition. It’s the data science equivalent of Leap code.”

Problem Specification

Timeframe: 00:00:47 - 00:01:05
Duration: 18s
Speaker: Devin Valenciano

The housing price estimation challenge represents a complex machine learning problem with real-world applicability, requiring sophisticated data analysis, feature engineering, and model selection techniques.

Problem Components:

  • Training dataset - Historical housing data with price labels
  • Test dataset - Houses requiring price prediction
  • Sample submission - Required output format
  • Data description - Feature documentation and context

Competition Context

Timeframe: 00:00:38 - 00:00:47
Duration: 9s
Speaker: Devin Valenciano

The competitive environment provides authentic constraints and evaluation metrics, ensuring the demonstration reflects real-world development pressures and performance standards.

“And they put full competitions online for people to put submissions up and and try and solve difficult problems in data science.”

Agent Mode Live Demonstration

Initial Setup and Context

Timeframe: 00:01:06 - 00:01:41
Duration: 35s
Speaker: Devin Valenciano

The demonstration begins with minimal human preparation—only a prompt file and competition data files—emphasizing Agent Mode’s ability to work from minimal starting context.

“So you can see I’ve got a couple things set up here and that’s it. I’ve got a prompt that I’m going to use to prompt agent mode. And I’ve got the the files that I just showed you pre downloaded.”

Starting Materials:

  • Custom prompt file for Agent Mode guidance
  • Competition data files (train.csv, test.csv, data_description.txt, sample_submission.csv)
  • VS Code with Copilot Agent Mode enabled

Autonomous Problem Analysis

Timeframe: 00:01:43 - 00:02:30
Duration: 47s
Speaker: Devin Valenciano

Agent Mode immediately demonstrates human-like problem-solving behavior by systematically analyzing the competition requirements and data structure without explicit instruction.

“And it does something great right away. It says I’m going to go fetch this URL that has the actual competition data and it’s using the built in fetch tool to go ahead and do that.”

Autonomous Analysis Steps:

  1. URL Fetching - Automatic retrieval of competition description
  2. File Reading - Systematic examination of data files
  3. Data Sampling - Preview of dataset structure (first 10 lines)
  4. Context Building - Understanding problem requirements and constraints

Technical Implementation Workflow

Timeframe: 00:02:08 - 00:02:45
Duration: 37s
Speaker: Devin Valenciano

The human-like analytical approach showcases Agent Mode’s sophisticated reasoning patterns, mirroring how experienced data scientists approach new problems.

“It does exactly what a human would do, right? It goes and reads the description file and reads the whole thing. And it’s like, OK, I understand a little bit about what this data is looking like.”

VS Code Platform Advantages

Native Notebook Integration

Timeframe: 00:02:45 - 00:03:15
Duration: 30s
Speaker: Devin Valenciano

VS Code’s competitive advantage emerges through seamless integration between Agent Mode and Jupyter notebooks, creating a unified development environment for data science workflows.

“So for those who don’t know, VS Code has some really incredible built in native notebook support. And this is like a primary tool among data scientists in the field.”

Integration Benefits:

  • Automatic notebook creation - Agent Mode generates appropriate project structure
  • Kernel management - Intelligent Python environment selection
  • Cell execution control - User permission system for code safety
  • Professional workflows - Industry-standard data science practices

Model Selection Capabilities

Timeframe: 00:03:22 - 00:03:47
Duration: 25s
Speaker: Devin Valenciano

The demonstration highlights VS Code Copilot’s multi-model support, allowing users to select optimal AI models for specific problem types and personal preferences.

“So right now we’re using the clod 3.7 sonnet model. I love clod 3.7 sonnet. It’s my favorite of all the ones listed here, but we have plenty to choose from within Copilot.”

Available Models:

  • Claude 3.5 Sonnet - Advanced reasoning capabilities
  • Claude 3.7 Sonnet - Preferred for deep analysis (demonstrated)
  • GPT-4 Turbo - Microsoft partnership integration
  • GPT-4.1-O - Latest OpenAI model
  • Gemini - Google model (preview availability)

Built-in Tool Ecosystem

Timeframe: 00:04:25 - 00:04:49
Duration: 24s
Speaker: Devin Valenciano

The prompt file system represents a significant productivity enhancement, enabling reusable development patterns and knowledge capture across projects.

“So this allows you to save custom prompts that can be used across multiple projects that can be tied to AVS code profile.”

Tool Ecosystem Features:

  • Prompt file storage - Reusable instruction patterns
  • Profile integration - Personalized development workflows
  • Cross-project application - Consistent approaches across domains
  • Efficiency optimization - Elimination of repetitive setup tasks

Technical Deep Dive: Data Science Pipeline

Environment Setup and Package Management

Timeframe: 00:07:22 - 00:08:00
Duration: 38s
Speaker: Devin Valenciano

Agent Mode demonstrates sophisticated dependency management by autonomously selecting and installing appropriate packages for the machine learning challenge without explicit guidance.

“So this just installs all the required packages. I didn’t tell it which packages to go install. It decided based on the problem set to which packages would be most useful in solving the Cagle competition.”

Automatically Selected Packages:

  • Data Manipulation: pandas, numpy
  • Visualization: matplotlib, seaborn
  • Machine Learning: scikit-learn (multiple modules)
  • Advanced ML: XGBoost
  • Statistical Analysis: scipy

Exploratory Data Analysis

Timeframe: 00:08:59 - 00:10:05
Duration: 1m 6s
Speaker: Devin Valenciano

The systematic data exploration phase reveals Agent Mode’s understanding of data science best practices, including data quality assessment and distribution analysis.

“And so next it jumps right into the exploratory data analysis. So it, it does what any human would do and it’s, it reads in the data and it starts to take a look at what the data actually is and what it can be used for.”

Analysis Components:

  • Data Loading - Systematic file reading and structure examination
  • Missing Value Detection - Quality assessment and data completeness
  • Distribution Analysis - Statistical property examination
  • Variable Correlation - Feature relationship identification

Data Preprocessing and Feature Engineering

Timeframe: 00:11:51 - 00:12:04
Duration: 13s
Speaker: Devin Valenciano

The preprocessing phase showcases advanced data science techniques, including log transformation for data normalization and categorical variable encoding.

“So it’s handling missing values, it’s transforming categorical variables, so variables that are, you know, in a string format into something that’s more usable for these machine learning models.”

86-Line Data Cleansing Implementation:

  • Missing value imputation - Strategic handling of incomplete data
  • Categorical encoding - String-to-numerical conversion
  • Feature scaling - Normalization for model optimization
  • Data validation - Quality assurance across datasets

Model Training and Evaluation

Timeframe: 00:12:26 - 00:13:20
Duration: 54s
Speaker: Devin Valenciano

The multi-algorithm approach demonstrates sophisticated machine learning strategy, training seven different models with systematic performance evaluation.

Basic Model Suite:

  • Linear Regression - Statistical baseline approach
  • Ridge Regression - Regularized linear model
  • Lasso Regression - Feature selection through regularization
  • Elastic Net - Combined Ridge/Lasso approach

Advanced Model Suite:

  • Random Forest - Ensemble tree-based method
  • Gradient Boosting - Sequential improvement algorithm
  • XGBoost - Optimized gradient boosting framework

Advanced Ensemble Methods

Timeframe: 00:13:37 - 00:14:01
Duration: 24s
Speaker: Devin Valenciano

The ensemble creation represents professional-grade machine learning technique, combining multiple model predictions for improved accuracy and robustness.

“But it actually does something even more advanced where it takes the the, the seven models that were generated and all those sets and it creates what’s called an ensemble.”

Ensemble Benefits:

  • Improved accuracy - Leveraging strengths of multiple approaches
  • Reduced overfitting - Balancing individual model limitations
  • Robust predictions - Consistent performance across data variations
  • Competition-grade technique - Industry-standard advanced methodology

Performance Results and Metrics

Competition Results

Timeframe: 00:14:41 - 00:15:01
Duration: 20s
Speaker: Devin Valenciano

The quantified success metrics provide concrete evidence of Agent Mode’s effectiveness, achieving professional-level results in a competitive environment.

“And so I got a score of 14,000 on this, which equates to about 300th place out of 6000. And submissions… So top 5% generated in about 10 minutes time with very little, you know, serious development on my end.”

Final Performance Metrics:

  • RMSE Score: 14,000
  • Competition Ranking: 300th out of 6,000 submissions
  • Percentile Performance: Top 5%
  • Development Time: Approximately 10 minutes
  • Human Intervention: Single prompt + execution permissions

Development Time Analysis

Timeframe: 00:07:35 - 00:07:43
Duration: 8s
Speaker: Devin Valenciano

The time efficiency comparison illustrates the dramatic acceleration in development workflows, from hours or days to minutes for complete data science pipelines.

“Even if I don’t know exactly what library does which thing I can go then research. It gives me a starting point to then go learn more efficiently.”

Efficiency Gains:

  • Traditional Approach: Hours to days for complete solution
  • Agent Mode Approach: 10 minutes for competition-ready submission
  • Setup Elimination: Automatic package selection and environment configuration
  • Learning Acceleration: Immediate exposure to expert-level techniques

Learning Efficiency Gains

Timeframe: 00:02:30 - 00:02:39
Duration: 9s
Speaker: Devin Valenciano

The educational value emerges through observation of professional development patterns, enabling rapid skill acquisition through practical application.

“It’s just saving me a ton of time and, and actually probably preventing me from learning a little bit, but it’s a pretty learning tool to, to, to work alongside.”

Limitations and Considerations

Non-Deterministic Behavior

Timeframe: 00:04:49 - 00:05:02
Duration: 13s
Speaker: Devin Valenciano

The session candidly addresses Agent Mode’s limitations, including unpredictable execution patterns that require human oversight and intervention.

“And unfortunately with agentic development flow, we have these little moments where it’s like, ah, shoot, I know this should work a certain way, but it’s not quite deterministic.”

Practical Challenges:

  • Unpredictable pausing - Agent may require continuation prompts
  • Execution variability - Non-deterministic workflow progression
  • Human supervision - Need for guidance during complex operations
  • Backup strategies - Importance of fallback plans for demonstrations

Ethical Usage Guidelines

Timeframe: 00:14:34 - 00:14:41
Duration: 7s
Speaker: Devin Valenciano

Clear ethical boundaries establish appropriate use cases for Agent Mode, emphasizing learning and development over competitive advantage in existing systems.

“And again, I don’t recommend that people are, you know, submitting Cagle submissions. And this isn’t the right way to for this to be done, but it’s a really quick way to learn.”

Best Practices

Timeframe: 00:10:05 - 00:10:12
Duration: 7s
Speaker: Devin Valenciano

The demonstration emphasizes the importance of user control and permission systems to prevent unauthorized or malicious code execution.

“We don’t want an agentic mode to just keep spinning and spinning and and eventually spending compute that it doesn’t need to compute.”

Recommended Approaches:

  • Learning and education - Primary use case for skill development
  • Prototyping and exploration - Rapid proof-of-concept creation
  • Starting point generation - Foundation code for human refinement
  • Pattern learning - Understanding professional development workflows

Educational Impact and Learning Acceleration

Knowledge Transfer Mechanisms

Timeframe: 00:11:04 - 00:11:08
Duration: 4s
Speaker: Devin Valenciano

Agent Mode serves as an educational accelerator by exposing users to advanced techniques and best practices they might not discover independently.

“But with agent mode, I can start to learn about these topics a lot faster.”

Learning Benefits:

  • Pattern Recognition - Observing expert-level problem-solving approaches
  • Technique Exposure - Introduction to advanced methodologies
  • Best Practice Integration - Learning industry-standard workflows
  • Rapid Iteration - Quick experimentation with different approaches

Skill Development Opportunities

Timeframe: 00:10:51 - 00:11:04
Duration: 13s
Speaker: Devin Valenciano

The demonstration highlights how Agent Mode enables exploration of advanced concepts that would typically require extensive domain expertise to discover and implement.

“And so it’s, it’s looking at all these different facets of the data that I personally would never think to explore because I’m not a professional data scientist.”

Educational Applications:

  • Data Science Methodology - Complete analytical workflow understanding
  • Feature Engineering - Variable transformation and selection techniques
  • Model Selection - Algorithm comparison and evaluation methods
  • Performance Optimization - Advanced techniques like ensemble methods

Appendix: Technical Implementation Details

Prompt File Configuration

The session references a custom prompt file system that enables reusable development patterns. While the specific prompt content isn’t detailed in the transcript, the system allows for:

  • Consistent Instructions - Standardized approach across projects
  • Profile Integration - Personalized development workflows
  • Cross-Project Reuse - Efficient knowledge transfer between domains

Model Performance Comparison

The automatic model selection and comparison process included systematic evaluation of seven different algorithms:

Traditional Statistical Methods:

  • Linear Regression (baseline)
  • Ridge Regression (L2 regularization)
  • Lasso Regression (L1 regularization)
  • Elastic Net (combined regularization)

Advanced Machine Learning:

  • Random Forest (ensemble tree-based)
  • Gradient Boosting (sequential optimization)
  • XGBoost (optimized gradient boosting)

Data Quality Assessment

The exploratory data analysis phase included comprehensive data quality evaluation:

  • Missing Value Analysis - Systematic identification of incomplete data
  • Distribution Examination - Statistical property assessment and visualization
  • Correlation Matrix - Feature relationship analysis for variable selection
  • Transformation Requirements - Log transformation for data normalization

References

Agent Mode and VS Code Documentation

  • VS Code Copilot Documentation - Comprehensive guide to Copilot features including Agent Mode capabilities, setup instructions, and best practices for autonomous coding assistance. Essential for understanding the technical foundation of the demonstrated capabilities.

  • Jupyter Notebook Support in VS Code - Detailed documentation on VS Code’s native notebook integration, which provides the foundation for Agent Mode’s data science workflow capabilities demonstrated in the session.

Data Science and Machine Learning Resources

  • Kaggle Learn - Free micro-courses in data science and machine learning that provide foundational knowledge for understanding the techniques Agent Mode applied in the housing price prediction challenge.

  • Scikit-learn Documentation - Comprehensive documentation for the machine learning library that Agent Mode automatically selected and utilized for model training and evaluation in the demonstration.

  • XGBoost Documentation - Technical documentation for the advanced gradient boosting library that Agent Mode autonomously incorporated into the ensemble model approach.

Competitive Programming and Data Science

  • Kaggle Housing Prices Competition - The actual competition referenced in the demonstration, providing context for the problem complexity and competitive benchmarks achieved by Agent Mode.

  • Machine Learning Ensemble Methods - Technical explanation of ensemble methods like the one Agent Mode automatically implemented, relevant for understanding the sophisticated approach demonstrated.

AI Model Integration

  • Claude Model Documentation - Understanding Claude 3.7 Sonnet capabilities and optimal usage patterns, which was the preferred model used in the demonstration for its deep analysis capabilities.

  • GitHub Copilot Best Practices - Best practices for effective AI pair programming that complement the autonomous capabilities demonstrated in Agent Mode.

Microsoft Build 2025 Context

  • Microsoft Build 2025 Sessions - Complete session catalog from Build 2025 conference, providing broader context for the developer tools and AI capabilities being introduced across Microsoft’s ecosystem.

  • VS Code Team Updates - Regular updates on VS Code feature releases and improvements, including Agent Mode development and related AI-powered development tools.

This session demonstrates the transformative potential of Agent Mode in democratizing advanced development capabilities, enabling developers and non-developers alike to achieve professional-grade results through autonomous AI assistance. The real-world Kaggle competition success validates the practical applicability of agentic development tools in solving complex, multi-step problems with minimal human intervention.